<!DOCTYPE html>
<html>
<head>
  <meta charset="utf-8">
  <!-- Global site tag (gtag.js) - Google Analytics -->
<script async src="https://www.googletagmanager.com/gtag/js?id=UA-169911533-1"></script>
<script>
  window.dataLayer = window.dataLayer || [];
  function gtag(){dataLayer.push(arguments);}
  gtag('js', new Date());

  gtag('config', 'UA-169911533-1');
</script>

  

  
  <title>Hadoop入门案例（一）: WordCount | 空城盼故人</title>
  <meta name="viewport" content="width=device-width, initial-scale=1, maximum-scale=1">
  <meta name="google-site-verification" content="NOVpqBhVVjZHOamocUf3Grijt7xAK_Engoe_FNrxE3Q" />
  <meta name="baidu-site-verification" content="HBrapLOndy" />
  
  <meta name="keywords" content="许嵩老公" />
  
  <meta name="description" content="MapReduce作业的输入与输出MapRecude计算框架是在键值对&lt;key, value&gt;上进行操作的。MapReduce计算框架将作业的输入视为一组&lt;key，value&gt;对，并生成一组&lt;key, value&gt;对作为其输出，可能是不同类型的。&lt;key, value&gt;中：  key和value的类都要由框架实现序列化，所以都需要实现org.apac">
<meta property="og:type" content="article">
<meta property="og:title" content="Hadoop入门案例（一）: WordCount">
<meta property="og:url" content="https://xiaoyan94.github.io/2020/06/26/First-MapReduce-example-WordCount/index.html">
<meta property="og:site_name" content="空城盼故人">
<meta property="og:description" content="MapReduce作业的输入与输出MapRecude计算框架是在键值对&lt;key, value&gt;上进行操作的。MapReduce计算框架将作业的输入视为一组&lt;key，value&gt;对，并生成一组&lt;key, value&gt;对作为其输出，可能是不同类型的。&lt;key, value&gt;中：  key和value的类都要由框架实现序列化，所以都需要实现org.apac">
<meta property="og:locale" content="zh_CN">
<meta property="article:published_time" content="2020-06-26T09:01:36.000Z">
<meta property="article:modified_time" content="2020-06-27T13:25:04.295Z">
<meta property="article:author" content="许嵩老公">
<meta property="article:tag" content="Hadoop">
<meta property="article:tag" content="MapReduce">
<meta name="twitter:card" content="summary">
  
    <link rel="alternate" href="../../../../atom.xml" title="空城盼故人" type="application/atom+xml">
  
  
    <link rel="icon" href="https://q1.qlogo.cn/g?b=qq&nk=979727728&s=640">
  
  
    <link href="//fonts.googleapis.com/css?family=Source+Code+Pro" rel="stylesheet" type="text/css">
  
  
<link rel="stylesheet" href="../../../../css/style.css">

  
<link rel="stylesheet" href="../../../../css/highlight.css">

<meta name="generator" content="Hexo 4.2.1"></head>

<body>
  <div id="fullpage" class="mobile-nav-right">
    <div class="fixed"></div>
    
      <div id="wrapper">
    
    
      <header id="header">
  <div id="nav-toggle" class="nav-toggle"></div>
  <div class="head-box global-width">
    <nav class="nav-box nav-right">
      
        <a class="nav-item" href="../../../../index.html" title
        
        >首页</a>
      
        <a class="nav-item" href="../../../../archives" title
        
        >归档</a>
      
        <a class="nav-item" href="../../../../quick-notes" title
        
        >小抄</a>
      
        <a class="nav-item" href="../../../../about" title
        
        >关于</a>
      
    </nav>
  </div>
</header>
      <div id="middlecontent" title class="global-width sidebar-right">
        <section id="main"><article id="post-First-MapReduce-example-WordCount" class="article global-container article-type-post" itemscope itemprop="blogPost">
  
    <header class="article-header">
      
  
    <h1 class="article-title" itemprop="name">
      Hadoop入门案例（一）: WordCount
    </h1>
  

    </header>
  
  <div class="article-meta">
    <a href="" class="article-date">
  <time datetime="2020-06-26T09:01:36.000Z" itemprop="datePublished">2020-06-26</time>
</a>
    
    
  <ul class="article-tag-list" itemprop="keywords"><li class="article-tag-list-item"><a class="article-tag-list-link" href="../../../../tags/Hadoop/" rel="tag">Hadoop</a></li><li class="article-tag-list-item"><a class="article-tag-list-link" href="../../../../tags/MapReduce/" rel="tag">MapReduce</a></li></ul>

  </div>
  
    <span id="busuanzi_container_page_pv">
      本文总阅读量<span id="busuanzi_value_page_pv"></span>次
    </span>
  

  <div class="article-inner">
    
    <div class="article-content article-content-cloud" itemprop="articleBody">
      
        
        <h2 id="MapReduce作业的输入与输出"><a href="#MapReduce作业的输入与输出" class="headerlink" title="MapReduce作业的输入与输出"></a>MapReduce作业的输入与输出</h2><p>MapRecude计算框架是在键值对&lt;key, value&gt;上进行操作的。MapReduce计算框架将作业的输入视为一组&lt;key，value&gt;对，并生成一组&lt;key, value&gt;对作为其输出，可能是不同类型的。&lt;key, value&gt;中：</p>
<ul>
<li>key和value的类都要由框架实现序列化，所以都需要实现<code>org.apache.hadoop.io.Writable</code>接口；</li>
<li>除此之外key的类还需要实现<code>org.apache.hadoop.io.WritableComparable</code>接口，因为在map操作之后还需要对key进行排序操作。</li>
</ul>
<p>MapReduce作业的输入和输出类型：</p>
<blockquote>
<p>(input) &lt;k1, v1&gt; -&gt; <strong>map</strong> -&gt; &lt;k2, v2&gt; -&gt; <strong>combine</strong> -&gt; &lt;k2, v2&gt; -&gt; <strong>reduce</strong> -&gt; &lt;k3, v3&gt; (output)</p>
</blockquote>
<a id="more"></a>

<hr>
<h2 id="MapReduce入门程序——WordCount"><a href="#MapReduce入门程序——WordCount" class="headerlink" title="MapReduce入门程序——WordCount"></a>MapReduce入门程序——WordCount</h2><p>WordCount是一个简单的应用程序，可以计算给定输入数据集中每个单词的出现次数。</p>
<p>WordCountApp.java代码：</p>
<div class="highlight-box"autocomplete="off" autocorrect="off" autocapitalize="off" spellcheck="false" contenteditable="false"data-rel="JAVA"><figure class="iseeu highlight /java"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br><span class="line">39</span><br><span class="line">40</span><br><span class="line">41</span><br><span class="line">42</span><br><span class="line">43</span><br><span class="line">44</span><br><span class="line">45</span><br><span class="line">46</span><br><span class="line">47</span><br><span class="line">48</span><br><span class="line">49</span><br><span class="line">50</span><br><span class="line">51</span><br><span class="line">52</span><br><span class="line">53</span><br><span class="line">54</span><br><span class="line">55</span><br><span class="line">56</span><br><span class="line">57</span><br><span class="line">58</span><br><span class="line">59</span><br><span class="line">60</span><br><span class="line">61</span><br><span class="line">62</span><br><span class="line">63</span><br><span class="line">64</span><br><span class="line">65</span><br><span class="line">66</span><br><span class="line">67</span><br><span class="line">68</span><br><span class="line">69</span><br><span class="line">70</span><br><span class="line">71</span><br><span class="line">72</span><br><span class="line">73</span><br><span class="line">74</span><br><span class="line">75</span><br><span class="line">76</span><br><span class="line">77</span><br><span class="line">78</span><br><span class="line">79</span><br><span class="line">80</span><br><span class="line">81</span><br><span class="line">82</span><br><span class="line">83</span><br><span class="line">84</span><br><span class="line">85</span><br><span class="line">86</span><br><span class="line">87</span><br><span class="line">88</span><br><span class="line">89</span><br><span class="line">90</span><br><span class="line">91</span><br><span class="line">92</span><br><span class="line">93</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">package</span> org.example;</span><br><span class="line"></span><br><span class="line"><span class="keyword">import</span> org.apache.hadoop.conf.Configuration;</span><br><span class="line"><span class="keyword">import</span> org.apache.hadoop.fs.Path;</span><br><span class="line"><span class="keyword">import</span> org.apache.hadoop.io.LongWritable;</span><br><span class="line"><span class="keyword">import</span> org.apache.hadoop.io.Text;</span><br><span class="line"><span class="keyword">import</span> org.apache.hadoop.mapreduce.Job;</span><br><span class="line"><span class="keyword">import</span> org.apache.hadoop.mapreduce.Mapper;</span><br><span class="line"><span class="keyword">import</span> org.apache.hadoop.mapreduce.Reducer;</span><br><span class="line"><span class="keyword">import</span> org.apache.hadoop.mapreduce.lib.input.FileInputFormat;</span><br><span class="line"><span class="keyword">import</span> org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;</span><br><span class="line"></span><br><span class="line"><span class="keyword">import</span> java.io.IOException;</span><br><span class="line"></span><br><span class="line"><span class="keyword">public</span> <span class="class"><span class="keyword">class</span> <span class="title">WordCountApp</span> </span>&#123;</span><br><span class="line"></span><br><span class="line">    <span class="comment">/**</span></span><br><span class="line"><span class="comment">     * Map：读取输入文件</span></span><br><span class="line"><span class="comment">     */</span></span><br><span class="line">    <span class="keyword">public</span> <span class="keyword">static</span> <span class="class"><span class="keyword">class</span> <span class="title">MyMapper</span> <span class="keyword">extends</span> <span class="title">Mapper</span>&lt;<span class="title">LongWritable</span>, <span class="title">Text</span>, <span class="title">Text</span>,</span></span><br><span class="line"><span class="class">            <span class="title">LongWritable</span>&gt; </span>&#123;</span><br><span class="line">        LongWritable one = <span class="keyword">new</span> LongWritable(<span class="number">1</span>);</span><br><span class="line"></span><br><span class="line">        <span class="meta">@Override</span></span><br><span class="line">        <span class="function"><span class="keyword">protected</span> <span class="keyword">void</span> <span class="title">map</span><span class="params">(LongWritable key, Text value, Context context)</span> <span class="keyword">throws</span> IOException, InterruptedException </span>&#123;</span><br><span class="line">            String line = value.toString(); <span class="comment">//每一行的数据</span></span><br><span class="line">            String[] words = line.split(<span class="string">" "</span>); <span class="comment">//按空格 分隔符拆分</span></span><br><span class="line">            <span class="keyword">for</span> (String word : words) &#123;</span><br><span class="line">                context.write(<span class="keyword">new</span> Text(word), one);</span><br><span class="line">            &#125;</span><br><span class="line">        &#125;</span><br><span class="line">    &#125;</span><br><span class="line"></span><br><span class="line">    <span class="comment">/**</span></span><br><span class="line"><span class="comment">     * Reduce：归并操作</span></span><br><span class="line"><span class="comment">     */</span></span><br><span class="line">    <span class="keyword">public</span> <span class="keyword">static</span> <span class="class"><span class="keyword">class</span> <span class="title">MyReducer</span> <span class="keyword">extends</span> <span class="title">Reducer</span>&lt;<span class="title">Text</span>, <span class="title">LongWritable</span>, <span class="title">Text</span>,</span></span><br><span class="line"><span class="class">            <span class="title">LongWritable</span>&gt; </span>&#123;</span><br><span class="line">        <span class="meta">@Override</span></span><br><span class="line">        <span class="function"><span class="keyword">protected</span> <span class="keyword">void</span> <span class="title">reduce</span><span class="params">(Text key, Iterable&lt;LongWritable&gt; values, Context context)</span> <span class="keyword">throws</span> IOException, InterruptedException </span>&#123;</span><br><span class="line"></span><br><span class="line">            <span class="keyword">long</span> sum = <span class="number">0</span>;</span><br><span class="line">            <span class="keyword">for</span> (LongWritable value :</span><br><span class="line">                    values) &#123;</span><br><span class="line">                <span class="comment">//求key总次数</span></span><br><span class="line">                sum += value.get();</span><br><span class="line">            &#125;</span><br><span class="line">            <span class="comment">// 输出此次reduce统计结果</span></span><br><span class="line">            context.write(key, <span class="keyword">new</span> LongWritable(sum));</span><br><span class="line">        &#125;</span><br><span class="line">    &#125;</span><br><span class="line"></span><br><span class="line">    <span class="comment">/**</span></span><br><span class="line"><span class="comment">     * 定义Driver：</span></span><br><span class="line"><span class="comment">     *</span></span><br><span class="line"><span class="comment">     * <span class="doctag">@param</span> args 第一个参数是输入文件路径，第二个参数是输出文件路径</span></span><br><span class="line"><span class="comment">     */</span></span><br><span class="line">    <span class="function"><span class="keyword">public</span> <span class="keyword">static</span> <span class="keyword">void</span> <span class="title">main</span><span class="params">(String[] args)</span> <span class="keyword">throws</span> IOException, ClassNotFoundException, InterruptedException </span>&#123;</span><br><span class="line"></span><br><span class="line">        Configuration configuration = <span class="keyword">new</span> Configuration();</span><br><span class="line"></span><br><span class="line">        <span class="comment">// 删除已存在的输出目录</span></span><br><span class="line">        Path outPath = <span class="keyword">new</span> Path(args[<span class="number">1</span>]);</span><br><span class="line">        FileSystem fs = FileSystem.get(configuration);</span><br><span class="line">        <span class="keyword">if</span> (fs.exists(outPath))&#123;</span><br><span class="line">            fs.delete(outPath, <span class="keyword">true</span>);</span><br><span class="line">            System.out.println(<span class="string">"output file exists, but is has been deleted"</span>);</span><br><span class="line">        &#125;</span><br><span class="line"></span><br><span class="line">        Job job = Job.getInstance(configuration, <span class="string">"wordcount"</span>);</span><br><span class="line"></span><br><span class="line">        <span class="comment">// 设置Job处理的类</span></span><br><span class="line">        job.setJarByClass(WordCountApp<span class="class">.<span class="keyword">class</span>)</span>;</span><br><span class="line"></span><br><span class="line">        <span class="comment">// 设置作业处理的输入路径</span></span><br><span class="line">        FileInputFormat.setInputPaths(job, <span class="keyword">new</span> Path(args[<span class="number">0</span>]));</span><br><span class="line"></span><br><span class="line">        <span class="comment">// 设置map相关参数</span></span><br><span class="line">        job.setMapperClass(MyMapper<span class="class">.<span class="keyword">class</span>)</span>;</span><br><span class="line">        job.setMapOutputKeyClass(Text<span class="class">.<span class="keyword">class</span>)</span>;</span><br><span class="line">        job.setMapOutputValueClass(LongWritable<span class="class">.<span class="keyword">class</span>)</span>;</span><br><span class="line"></span><br><span class="line">        <span class="comment">// 设置reduce相关参数</span></span><br><span class="line">        job.setReducerClass(MyReducer<span class="class">.<span class="keyword">class</span>)</span>;</span><br><span class="line">        job.setOutputKeyClass(Text<span class="class">.<span class="keyword">class</span>)</span>;</span><br><span class="line">        job.setOutputValueClass(LongWritable<span class="class">.<span class="keyword">class</span>)</span>;</span><br><span class="line"></span><br><span class="line">        <span class="comment">// 设置作业处理的输出路径</span></span><br><span class="line">        FileOutputFormat.setOutputPath(job, <span class="keyword">new</span> Path(args[<span class="number">1</span>]));</span><br><span class="line"></span><br><span class="line">        System.exit(job.waitForCompletion(<span class="keyword">true</span>) ? <span class="number">0</span> : <span class="number">1</span>);</span><br><span class="line">    &#125;</span><br><span class="line">&#125;</span><br></pre></td></tr></table></figure></div>

<h3 id="WordCount代码分析"><a href="#WordCount代码分析" class="headerlink" title="WordCount代码分析"></a>WordCount代码分析</h3><p>Mapper的实现类如下：在map方法中，一次处理一行的数据，由<code>TextInputFormat</code>指定，它将一行字符串以空格为分隔符拆分成单词，并输出 <code>单词-次数</code> 键值对 <code>&lt;&lt;word&gt;,1&gt;</code></p>
<div class="highlight-box"autocomplete="off" autocorrect="off" autocapitalize="off" spellcheck="false" contenteditable="false"data-rel="JAVA"><figure class="iseeu highlight /java"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">public</span> <span class="keyword">static</span> <span class="class"><span class="keyword">class</span> <span class="title">MyMapper</span> <span class="keyword">extends</span> <span class="title">Mapper</span>&lt;<span class="title">LongWritable</span>, <span class="title">Text</span>, <span class="title">Text</span>,</span></span><br><span class="line"><span class="class">        <span class="title">LongWritable</span>&gt; </span>&#123;</span><br><span class="line">    LongWritable one = <span class="keyword">new</span> LongWritable(<span class="number">1</span>);</span><br><span class="line"></span><br><span class="line">    <span class="meta">@Override</span></span><br><span class="line">    <span class="function"><span class="keyword">protected</span> <span class="keyword">void</span> <span class="title">map</span><span class="params">(LongWritable key, Text value, Context context)</span> <span class="keyword">throws</span> IOException, InterruptedException </span>&#123;</span><br><span class="line">        String line = value.toString(); <span class="comment">//每一行的数据</span></span><br><span class="line">        String[] words = line.split(<span class="string">" "</span>); <span class="comment">//按空格 分隔符拆分</span></span><br><span class="line">        <span class="keyword">for</span> (String word : words) &#123;</span><br><span class="line">            context.write(<span class="keyword">new</span> Text(word), one);</span><br><span class="line">        &#125;</span><br><span class="line">    &#125;</span><br><span class="line">&#125;</span><br></pre></td></tr></table></figure></div>

<p>Reducer的实现类如下：在reduce方法中，只是对values进行求和，这些values是每个key的出现次数（在本示例中单词出现的次数）。</p>
<div class="highlight-box"autocomplete="off" autocorrect="off" autocapitalize="off" spellcheck="false" contenteditable="false"data-rel="JAVA"><figure class="iseeu highlight /java"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">public</span> <span class="keyword">static</span> <span class="class"><span class="keyword">class</span> <span class="title">MyReducer</span> <span class="keyword">extends</span> <span class="title">Reducer</span>&lt;<span class="title">Text</span>, <span class="title">LongWritable</span>, <span class="title">Text</span>,</span></span><br><span class="line"><span class="class">        <span class="title">LongWritable</span>&gt; </span>&#123;</span><br><span class="line">    <span class="meta">@Override</span></span><br><span class="line">    <span class="function"><span class="keyword">protected</span> <span class="keyword">void</span> <span class="title">reduce</span><span class="params">(Text key, Iterable&lt;LongWritable&gt; values, Context context)</span> <span class="keyword">throws</span> IOException, InterruptedException </span>&#123;</span><br><span class="line"></span><br><span class="line">        <span class="keyword">long</span> sum = <span class="number">0</span>;</span><br><span class="line">        <span class="keyword">for</span> (LongWritable value :</span><br><span class="line">                values) &#123;</span><br><span class="line">            <span class="comment">//求key总次数</span></span><br><span class="line">            sum += value.get();</span><br><span class="line">        &#125;</span><br><span class="line">        <span class="comment">// 输出此次reduce统计结果</span></span><br><span class="line">        context.write(key, <span class="keyword">new</span> LongWritable(sum));</span><br><span class="line">    &#125;</span><br><span class="line">&#125;</span><br></pre></td></tr></table></figure></div>

<p>最后，在main方法指定作业的各个方面，例如作业中的输入&#x2F;输出路径(通过命令行传递)、键&#x2F;值类型、输入&#x2F;输出格式等。 然后，它调用<code>job.waitForCompletion</code>方法来提交作业并监视其进度。</p>
<h3 id="提交作业"><a href="#提交作业" class="headerlink" title="提交作业"></a>提交作业</h3><p>将写好的程序提交到YARN执行：</p>
<ol>
<li><p>因为使用Maven搭建，在项目根目录下执行命令<code>mvn clean package -DskipTests</code>打包</p>
<div class="highlight-box"autocomplete="off" autocorrect="off" autocapitalize="off" spellcheck="false" contenteditable="false"data-rel="BASH"><figure class="iseeu highlight /bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br></pre></td><td class="code"><pre><span class="line">$ mvn clean package -DskipTests</span><br><span class="line"> [INFO] Scanning <span class="keyword">for</span> projects...</span><br><span class="line"> [INFO]</span><br><span class="line"> [INFO] ----------------------&lt; org.example:hadoop-train &gt;----------------------</span><br><span class="line"> [INFO] Building hadoop-train 1.0</span><br><span class="line"> [INFO] --------------------------------[ jar ]---------------------------------</span><br><span class="line"> [INFO]</span><br><span class="line"> [INFO] --- maven-clean-plugin:3.1.0:clean (default-clean) @ hadoop-train ---</span><br><span class="line"> [INFO] Deleting /Users/yan/IdeaProjects/com.xxx.hadoop/com.xxx.hadoop/target</span><br><span class="line"> [INFO]</span><br><span class="line"> [INFO] --- maven-resources-plugin:3.0.2:resources (default-resources) @ hadoop-train ---</span><br><span class="line"> [INFO] Using <span class="string">'UTF-8'</span> encoding to copy filtered resources.</span><br><span class="line"> [INFO] skip non existing resourceDirectory /Users/yan/IdeaProjects/com.xxx.hadoop/com.xxx.hadoop/src/main/resources</span><br><span class="line"> [INFO]</span><br><span class="line"> [INFO] --- maven-compiler-plugin:3.8.0:compile (default-compile) @ hadoop-train ---</span><br><span class="line"> [INFO] Changes detected - recompiling the module!</span><br><span class="line"> [INFO] Compiling 5 <span class="built_in">source</span> files to /Users/yan/IdeaProjects/com.xxx.hadoop/com.xxx.hadoop/target/classes</span><br><span class="line"> [INFO]</span><br><span class="line"> [INFO] --- maven-resources-plugin:3.0.2:testResources (default-testResources) @ hadoop-train ---</span><br><span class="line"> [INFO] Using <span class="string">'UTF-8'</span> encoding to copy filtered resources.</span><br><span class="line"> [INFO] skip non existing resourceDirectory /Users/yan/IdeaProjects/com.xxx.hadoop/com.xxx.hadoop/src/<span class="built_in">test</span>/resources</span><br><span class="line"> [INFO]</span><br><span class="line"> [INFO] --- maven-compiler-plugin:3.8.0:testCompile (default-testCompile) @ hadoop-train ---</span><br><span class="line"> [INFO] Changes detected - recompiling the module!</span><br><span class="line"> [INFO] Compiling 1 <span class="built_in">source</span> file to /Users/yan/IdeaProjects/com.xxx.hadoop/com.xxx.hadoop/target/<span class="built_in">test</span>-classes</span><br><span class="line"> [INFO]</span><br><span class="line"> [INFO] --- maven-surefire-plugin:2.22.1:<span class="built_in">test</span> (default-test) @ hadoop-train ---</span><br><span class="line"> [INFO] Tests are skipped.</span><br><span class="line"> [INFO]</span><br><span class="line"> [INFO] --- maven-jar-plugin:3.0.2:jar (default-jar) @ hadoop-train ---</span><br><span class="line"> [INFO] Building jar: /Users/yan/IdeaProjects/com.xxx.hadoop/com.xxx.hadoop/target/hadoop-train-1.0.jar</span><br><span class="line"> [INFO] ------------------------------------------------------------------------</span><br><span class="line"> [INFO] BUILD SUCCESS</span><br><span class="line"> [INFO] ------------------------------------------------------------------------</span><br><span class="line"> [INFO] Total time:  2.800 s</span><br><span class="line"> [INFO] Finished at: 2020-06-24T20:17:52+08:00</span><br><span class="line"> [INFO] ------------------------------------------------------------------------</span><br></pre></td></tr></table></figure></div>
</li>
<li><p>使用命令<code>scp -P26885 target/hadoop-train-1.0.jar root@23.105.206.170:~/hadoop/lib</code>将打包好的jar文件上传至远程服务器~&#x2F;hadoop&#x2F;lib目录下：</p>
<div class="highlight-box"autocomplete="off" autocorrect="off" autocapitalize="off" spellcheck="false" contenteditable="false"data-rel="BASH"><figure class="iseeu highlight /bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line">$ scp -P26885 target/hadoop-train-1.0.jar root@23.105.206.170:~/hadoop/lib</span><br><span class="line">hadoop-train-1.0.jar     100%   17KB   6.7KB/s   00:02</span><br></pre></td></tr></table></figure></div>
</li>
<li><p>以如下流程在服务器上执行上传的程序：</p>
<div class="highlight-box"autocomplete="off" autocorrect="off" autocapitalize="off" spellcheck="false" contenteditable="false"data-rel="BASH"><figure class="iseeu highlight /bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br></pre></td><td class="code"><pre><span class="line">$ hadoop fs -put in.txt /in.txt <span class="comment"># 将in.txt上传至HDFS根目录下</span></span><br><span class="line">...</span><br><span class="line">$ hadoop jar ~/hadoop/lib/hadoop-train-1.0.jar org.example.WordCountApp /in.txt /mymprd/wordcount <span class="comment"># 运行WordCount程序</span></span><br><span class="line">...</span><br><span class="line">$ hadoop fs -cat /mymprd/wordcount/part-r-00000 <span class="comment">#查看程序输出结果</span></span><br><span class="line">...</span><br></pre></td></tr></table></figure></div>
</li>
<li><p>输入路径也可以是一个文件夹，文件夹内有多个输入文件：</p>
<div class="highlight-box"autocomplete="off" autocorrect="off" autocapitalize="off" spellcheck="false" contenteditable="false"data-rel="BASH"><figure class="iseeu highlight /bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br><span class="line">39</span><br><span class="line">40</span><br><span class="line">41</span><br><span class="line">42</span><br><span class="line">43</span><br><span class="line">44</span><br><span class="line">45</span><br><span class="line">46</span><br><span class="line">47</span><br><span class="line">48</span><br><span class="line">49</span><br><span class="line">50</span><br><span class="line">51</span><br><span class="line">52</span><br><span class="line">53</span><br><span class="line">54</span><br><span class="line">55</span><br><span class="line">56</span><br><span class="line">57</span><br><span class="line">58</span><br><span class="line">59</span><br><span class="line">60</span><br><span class="line">61</span><br><span class="line">62</span><br><span class="line">63</span><br><span class="line">64</span><br><span class="line">65</span><br><span class="line">66</span><br><span class="line">67</span><br><span class="line">68</span><br><span class="line">69</span><br><span class="line">70</span><br><span class="line">71</span><br><span class="line">72</span><br><span class="line">73</span><br><span class="line">74</span><br><span class="line">75</span><br><span class="line">76</span><br><span class="line">77</span><br><span class="line">78</span><br><span class="line">79</span><br><span class="line">80</span><br><span class="line">81</span><br><span class="line">82</span><br><span class="line">83</span><br><span class="line">84</span><br><span class="line">85</span><br><span class="line">86</span><br><span class="line">87</span><br><span class="line">88</span><br><span class="line">89</span><br><span class="line">90</span><br><span class="line">91</span><br><span class="line">92</span><br><span class="line">93</span><br></pre></td><td class="code"><pre><span class="line">root@brave-post-2:~/hadoop/script<span class="comment"># cat input/in1.txt # 准备输入文件in1、in2</span></span><br><span class="line">Hello World Bye World</span><br><span class="line">root@brave-post-2:~/hadoop/script<span class="comment"># cat input/in2.txt</span></span><br><span class="line">Hello Hadoop Goodbye Hadoop</span><br><span class="line">Hello Goodbye</span><br><span class="line">root@brave-post-2:~/hadoop/script<span class="comment"># hadoop fs -put input / # -put也能上传一个文件夹至HDFS</span></span><br><span class="line">20/06/26 08:05:08 WARN util.NativeCodeLoader: Unable to load native-hadoop library <span class="keyword">for</span> your platform... using <span class="built_in">builtin</span>-java classes <span class="built_in">where</span> applicable</span><br><span class="line">root@brave-post-2:~/hadoop/script<span class="comment"># hadoop fs -ls /input</span></span><br><span class="line">20/06/26 08:05:34 WARN util.NativeCodeLoader: Unable to load native-hadoop library <span class="keyword">for</span> your platform... using <span class="built_in">builtin</span>-java classes <span class="built_in">where</span> applicable</span><br><span class="line">Found 2 items</span><br><span class="line">-rw-r--r--   1 root supergroup         22 2020-06-26 08:05 /input/in1.txt</span><br><span class="line">-rw-r--r--   1 root supergroup         42 2020-06-26 08:05 /input/in2.txt</span><br><span class="line">root@brave-post-2:~/hadoop/script<span class="comment"># hadoop jar ~/hadoop/lib/hadoop-train-1.0.jar org.example.WordCount2App /input /mymprd/wordcount # 以input文件夹作为输入路径</span></span><br><span class="line">20/06/26 08:06:08 WARN util.NativeCodeLoader: Unable to load native-hadoop library <span class="keyword">for</span> your platform... using <span class="built_in">builtin</span>-java classes <span class="built_in">where</span> applicable</span><br><span class="line">output file exists, but is has deleted</span><br><span class="line">20/06/26 08:06:10 INFO client.RMProxy: Connecting to ResourceManager at /0.0.0.0:8032</span><br><span class="line">20/06/26 08:06:13 WARN mapreduce.JobResourceUploader: Hadoop <span class="built_in">command</span>-line option parsing not performed. Implement the Tool interface and execute your application with ToolRunner to remedy this.</span><br><span class="line">20/06/26 08:06:14 INFO input.FileInputFormat: Total input paths to process : 2</span><br><span class="line">20/06/26 08:06:14 INFO mapreduce.JobSubmitter: number of splits:2</span><br><span class="line">20/06/26 08:06:14 INFO mapreduce.JobSubmitter: Submitting tokens <span class="keyword">for</span> job: job_1593005599056_0021</span><br><span class="line">20/06/26 08:06:15 INFO impl.YarnClientImpl: Submitted application application_1593005599056_0021</span><br><span class="line">20/06/26 08:06:15 INFO mapreduce.Job: The url to track the job: http://localhost:8088/proxy/application_1593005599056_0021/</span><br><span class="line">20/06/26 08:06:15 INFO mapreduce.Job: Running job: job_1593005599056_0021</span><br><span class="line">20/06/26 08:06:30 INFO mapreduce.Job: Job job_1593005599056_0021 running <span class="keyword">in</span> uber mode : <span class="literal">false</span></span><br><span class="line">20/06/26 08:06:30 INFO mapreduce.Job:  map 0% reduce 0%</span><br><span class="line">20/06/26 08:06:48 INFO mapreduce.Job:  map 50% reduce 0%</span><br><span class="line">20/06/26 08:06:49 INFO mapreduce.Job:  map 100% reduce 0%</span><br><span class="line">20/06/26 08:07:00 INFO mapreduce.Job:  map 100% reduce 100%</span><br><span class="line">20/06/26 08:07:04 INFO mapreduce.Job: Job job_1593005599056_0021 completed successfully</span><br><span class="line">20/06/26 08:07:05 INFO mapreduce.Job: Counters: 49</span><br><span class="line">    File System Counters</span><br><span class="line">        FILE: Number of bytes <span class="built_in">read</span>=170</span><br><span class="line">        FILE: Number of bytes written=334434</span><br><span class="line">        FILE: Number of <span class="built_in">read</span> operations=0</span><br><span class="line">        FILE: Number of large <span class="built_in">read</span> operations=0</span><br><span class="line">        FILE: Number of write operations=0</span><br><span class="line">        HDFS: Number of bytes <span class="built_in">read</span>=274</span><br><span class="line">        HDFS: Number of bytes written=41</span><br><span class="line">        HDFS: Number of <span class="built_in">read</span> operations=9</span><br><span class="line">        HDFS: Number of large <span class="built_in">read</span> operations=0</span><br><span class="line">        HDFS: Number of write operations=2</span><br><span class="line">    Job Counters</span><br><span class="line">        Launched map tasks=2</span><br><span class="line">        Launched reduce tasks=1</span><br><span class="line">        Data-local map tasks=3</span><br><span class="line">        Total time spent by all maps <span class="keyword">in</span> occupied slots (ms)=31069</span><br><span class="line">        Total time spent by all reduces <span class="keyword">in</span> occupied slots (ms)=8852</span><br><span class="line">        Total time spent by all map tasks (ms)=31069</span><br><span class="line">        Total time spent by all reduce tasks (ms)=8852</span><br><span class="line">        Total vcore-seconds taken by all map tasks=31069</span><br><span class="line">        Total vcore-seconds taken by all reduce tasks=8852</span><br><span class="line">        Total megabyte-seconds taken by all map tasks=31814656</span><br><span class="line">        Total megabyte-seconds taken by all reduce tasks=9064448</span><br><span class="line">    Map-Reduce Framework</span><br><span class="line">        Map input records=3</span><br><span class="line">        Map output records=10</span><br><span class="line">        Map output bytes=144</span><br><span class="line">        Map output materialized bytes=176</span><br><span class="line">        Input split bytes=210</span><br><span class="line">        Combine input records=0</span><br><span class="line">        Combine output records=0</span><br><span class="line">        Reduce input groups=5</span><br><span class="line">        Reduce shuffle bytes=176</span><br><span class="line">        Reduce input records=10</span><br><span class="line">        Reduce output records=5</span><br><span class="line">        Spilled Records=20</span><br><span class="line">        Shuffled Maps =2</span><br><span class="line">        Failed Shuffles=0</span><br><span class="line">        Merged Map outputs=2</span><br><span class="line">        GC time elapsed (ms)=634</span><br><span class="line">        CPU time spent (ms)=3900</span><br><span class="line">        Physical memory (bytes) snapshot=472780800</span><br><span class="line">        Virtual memory (bytes) snapshot=7740059648</span><br><span class="line">        Total committed heap usage (bytes)=264744960</span><br><span class="line">    Shuffle Errors</span><br><span class="line">        BAD_ID=0</span><br><span class="line">        CONNECTION=0</span><br><span class="line">        IO_ERROR=0</span><br><span class="line">        WRONG_LENGTH=0</span><br><span class="line">        WRONG_MAP=0</span><br><span class="line">        WRONG_REDUCE=0</span><br><span class="line">    File Input Format Counters</span><br><span class="line">        Bytes Read=64</span><br><span class="line">    File Output Format Counters</span><br><span class="line">        Bytes Written=41</span><br><span class="line">root@brave-post-2:~/hadoop/script<span class="comment"># hadoop fs -cat /mymprd/wordcount/part-r-00000 # 查看程序执行结果</span></span><br><span class="line">20/06/26 08:07:27 WARN util.NativeCodeLoader: Unable to load native-hadoop library <span class="keyword">for</span> your platform... using <span class="built_in">builtin</span>-java classes <span class="built_in">where</span> applicable</span><br><span class="line">Bye 1</span><br><span class="line">Goodbye 2</span><br><span class="line">Hadoop 2</span><br><span class="line">Hello 3</span><br><span class="line">World 2</span><br><span class="line">root@brave-post-2:~/hadoop/script<span class="comment">#</span></span><br></pre></td></tr></table></figure></div></li>
</ol>
<hr>
<h3 id="使用Combiner降低map和reduce之间的数据传输量"><a href="#使用Combiner降低map和reduce之间的数据传输量" class="headerlink" title="使用Combiner降低map和reduce之间的数据传输量"></a>使用Combiner降低map和reduce之间的数据传输量</h3><div class="highlight-box"autocomplete="off" autocorrect="off" autocapitalize="off" spellcheck="false" contenteditable="false"data-rel="JAVA"><figure class="iseeu highlight /java"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br><span class="line">39</span><br><span class="line">40</span><br><span class="line">41</span><br><span class="line">42</span><br><span class="line">43</span><br><span class="line">44</span><br><span class="line">45</span><br><span class="line">46</span><br><span class="line">47</span><br><span class="line">48</span><br><span class="line">49</span><br><span class="line">50</span><br><span class="line">51</span><br><span class="line">52</span><br><span class="line">53</span><br><span class="line">54</span><br><span class="line">55</span><br><span class="line">56</span><br><span class="line">57</span><br><span class="line">58</span><br><span class="line">59</span><br><span class="line">60</span><br><span class="line">61</span><br><span class="line">62</span><br><span class="line">63</span><br><span class="line">64</span><br><span class="line">65</span><br><span class="line">66</span><br><span class="line">67</span><br><span class="line">68</span><br><span class="line">69</span><br><span class="line">70</span><br><span class="line">71</span><br><span class="line">72</span><br><span class="line">73</span><br><span class="line">74</span><br><span class="line">75</span><br><span class="line">76</span><br><span class="line">77</span><br><span class="line">78</span><br><span class="line">79</span><br><span class="line">80</span><br><span class="line">81</span><br><span class="line">82</span><br><span class="line">83</span><br><span class="line">84</span><br><span class="line">85</span><br><span class="line">86</span><br><span class="line">87</span><br><span class="line">88</span><br><span class="line">89</span><br><span class="line">90</span><br><span class="line">91</span><br><span class="line">92</span><br><span class="line">93</span><br><span class="line">94</span><br><span class="line">95</span><br><span class="line">96</span><br><span class="line">97</span><br><span class="line">98</span><br><span class="line">99</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">package</span> org.example;</span><br><span class="line"></span><br><span class="line"><span class="keyword">import</span> org.apache.hadoop.conf.Configuration;</span><br><span class="line"><span class="keyword">import</span> org.apache.hadoop.fs.FileSystem;</span><br><span class="line"><span class="keyword">import</span> org.apache.hadoop.fs.Path;</span><br><span class="line"><span class="keyword">import</span> org.apache.hadoop.io.LongWritable;</span><br><span class="line"><span class="keyword">import</span> org.apache.hadoop.io.Text;</span><br><span class="line"><span class="keyword">import</span> org.apache.hadoop.mapreduce.Job;</span><br><span class="line"><span class="keyword">import</span> org.apache.hadoop.mapreduce.Mapper;</span><br><span class="line"><span class="keyword">import</span> org.apache.hadoop.mapreduce.Reducer;</span><br><span class="line"><span class="keyword">import</span> org.apache.hadoop.mapreduce.lib.input.FileInputFormat;</span><br><span class="line"><span class="keyword">import</span> org.apache.hadoop.mapreduce.lib.input.TextInputFormat;</span><br><span class="line"><span class="keyword">import</span> org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;</span><br><span class="line"></span><br><span class="line"><span class="keyword">import</span> java.io.IOException;</span><br><span class="line"></span><br><span class="line"><span class="keyword">public</span> <span class="class"><span class="keyword">class</span> <span class="title">CombinerApp</span> </span>&#123;</span><br><span class="line"></span><br><span class="line">    <span class="comment">/**</span></span><br><span class="line"><span class="comment">     * Map：读取输入文件</span></span><br><span class="line"><span class="comment">     */</span></span><br><span class="line">    <span class="keyword">public</span> <span class="keyword">static</span> <span class="class"><span class="keyword">class</span> <span class="title">MyMapper</span> <span class="keyword">extends</span> <span class="title">Mapper</span>&lt;<span class="title">LongWritable</span>, <span class="title">Text</span>, <span class="title">Text</span>,</span></span><br><span class="line"><span class="class">            <span class="title">LongWritable</span>&gt; </span>&#123;</span><br><span class="line">        LongWritable one = <span class="keyword">new</span> LongWritable(<span class="number">1</span>);</span><br><span class="line"></span><br><span class="line">        <span class="meta">@Override</span></span><br><span class="line">        <span class="function"><span class="keyword">protected</span> <span class="keyword">void</span> <span class="title">map</span><span class="params">(LongWritable key, Text value, Context context)</span> <span class="keyword">throws</span> IOException, InterruptedException </span>&#123;</span><br><span class="line">            String line = value.toString(); <span class="comment">//每一行的数据</span></span><br><span class="line">            String[] words = line.split(<span class="string">" "</span>); <span class="comment">//按空格 分隔符拆分</span></span><br><span class="line">            <span class="keyword">for</span> (String word : words) &#123;</span><br><span class="line">                context.write(<span class="keyword">new</span> Text(word), one);</span><br><span class="line">            &#125;</span><br><span class="line">        &#125;</span><br><span class="line">    &#125;</span><br><span class="line"></span><br><span class="line">    <span class="comment">/**</span></span><br><span class="line"><span class="comment">     * Reduce：归并操作</span></span><br><span class="line"><span class="comment">     */</span></span><br><span class="line">    <span class="keyword">public</span> <span class="keyword">static</span> <span class="class"><span class="keyword">class</span> <span class="title">MyReducer</span> <span class="keyword">extends</span> <span class="title">Reducer</span>&lt;<span class="title">Text</span>, <span class="title">LongWritable</span>, <span class="title">Text</span>,</span></span><br><span class="line"><span class="class">            <span class="title">LongWritable</span>&gt; </span>&#123;</span><br><span class="line">        <span class="meta">@Override</span></span><br><span class="line">        <span class="function"><span class="keyword">protected</span> <span class="keyword">void</span> <span class="title">reduce</span><span class="params">(Text key, Iterable&lt;LongWritable&gt; values, Context context)</span> <span class="keyword">throws</span> IOException, InterruptedException </span>&#123;</span><br><span class="line"></span><br><span class="line">            <span class="keyword">long</span> sum = <span class="number">0</span>;</span><br><span class="line">            <span class="keyword">for</span> (LongWritable value :</span><br><span class="line">                    values) &#123;</span><br><span class="line">                <span class="comment">//求key总次数</span></span><br><span class="line">                sum += value.get();</span><br><span class="line">            &#125;</span><br><span class="line">            <span class="comment">// 输出此次reduce统计结果</span></span><br><span class="line">            context.write(key, <span class="keyword">new</span> LongWritable(sum));</span><br><span class="line">        &#125;</span><br><span class="line">    &#125;</span><br><span class="line"></span><br><span class="line">    <span class="comment">/**</span></span><br><span class="line"><span class="comment">     * 定义Driver：</span></span><br><span class="line"><span class="comment">     *</span></span><br><span class="line"><span class="comment">     * <span class="doctag">@param</span> args</span></span><br><span class="line"><span class="comment">     */</span></span><br><span class="line">    <span class="function"><span class="keyword">public</span> <span class="keyword">static</span> <span class="keyword">void</span> <span class="title">main</span><span class="params">(String[] args)</span> <span class="keyword">throws</span> IOException, ClassNotFoundException, InterruptedException </span>&#123;</span><br><span class="line"></span><br><span class="line">        Configuration configuration = <span class="keyword">new</span> Configuration();</span><br><span class="line"></span><br><span class="line">        <span class="comment">// 删除已存在的输出目录</span></span><br><span class="line">        Path outPath = <span class="keyword">new</span> Path(args[<span class="number">1</span>]);</span><br><span class="line">        FileSystem fs = FileSystem.get(configuration);</span><br><span class="line">        <span class="keyword">if</span> (fs.exists(outPath))&#123;</span><br><span class="line">            fs.delete(outPath, <span class="keyword">true</span>);</span><br><span class="line">            System.out.println(<span class="string">"output file exists, but is has deleted"</span>);</span><br><span class="line">        &#125;</span><br><span class="line"></span><br><span class="line">        Job job = Job.getInstance(configuration, <span class="string">"wordcount"</span>);</span><br><span class="line"></span><br><span class="line">        <span class="comment">// 设置Job处理的类</span></span><br><span class="line">        job.setJarByClass(CombinerApp<span class="class">.<span class="keyword">class</span>)</span>;</span><br><span class="line"></span><br><span class="line">        <span class="comment">// 设置作业处理的输入路径</span></span><br><span class="line">        FileInputFormat.setInputPaths(job, <span class="keyword">new</span> Path(args[<span class="number">0</span>]));</span><br><span class="line"></span><br><span class="line">        <span class="comment">// 设置map相关参数</span></span><br><span class="line">        job.setMapperClass(MyMapper<span class="class">.<span class="keyword">class</span>)</span>;</span><br><span class="line">        job.setMapOutputKeyClass(Text<span class="class">.<span class="keyword">class</span>)</span>;</span><br><span class="line">        job.setMapOutputValueClass(LongWritable<span class="class">.<span class="keyword">class</span>)</span>;</span><br><span class="line"></span><br><span class="line">        <span class="comment">// 设置reduce相关参数</span></span><br><span class="line">        job.setReducerClass(MyReducer<span class="class">.<span class="keyword">class</span>)</span>;</span><br><span class="line">        job.setOutputKeyClass(Text<span class="class">.<span class="keyword">class</span>)</span>;</span><br><span class="line">        job.setOutputValueClass(LongWritable<span class="class">.<span class="keyword">class</span>)</span>;</span><br><span class="line"></span><br><span class="line">        <span class="comment">// 设置Combiner处理类，相当于在Map之后先在本地进行一次合并之后再通过网络传输数据给Reduce Tasks</span></span><br><span class="line">        <span class="comment">// 使用场景：求次数；求和；  不能使用的场景：平均数</span></span><br><span class="line">        job.setCombinerClass(MyReducer<span class="class">.<span class="keyword">class</span>)</span>;</span><br><span class="line"></span><br><span class="line">        <span class="comment">// 设置作业处理的输出路径</span></span><br><span class="line">        FileOutputFormat.setOutputPath(job, <span class="keyword">new</span> Path(args[<span class="number">1</span>]));</span><br><span class="line"></span><br><span class="line">        System.exit(job.waitForCompletion(<span class="keyword">true</span>) ? <span class="number">0</span> : <span class="number">1</span>);</span><br><span class="line">    &#125;</span><br><span class="line">&#125;</span><br></pre></td></tr></table></figure></div>

<p>关于Combiner：设置Combiner处理类，相当于在<code>map</code>操作之后先在本地进行一次合并（即local aggregation）之后再通过网络传输数据给<code>reduce</code>。</p>
<p>不过combiner的使用是有场景限制的：比如求次数、求和可以用；但是求平均数就不能用。</p>
<hr>
<h3 id="和wordcount相似的partitioner"><a href="#和wordcount相似的partitioner" class="headerlink" title="和wordcount相似的partitioner"></a>和wordcount相似的partitioner</h3><p>PartitionerApp 代码：</p>
<div class="highlight-box"autocomplete="off" autocorrect="off" autocapitalize="off" spellcheck="false" contenteditable="false"data-rel="JAVA"><figure class="iseeu highlight /java"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br><span class="line">39</span><br><span class="line">40</span><br><span class="line">41</span><br><span class="line">42</span><br><span class="line">43</span><br><span class="line">44</span><br><span class="line">45</span><br><span class="line">46</span><br><span class="line">47</span><br><span class="line">48</span><br><span class="line">49</span><br><span class="line">50</span><br><span class="line">51</span><br><span class="line">52</span><br><span class="line">53</span><br><span class="line">54</span><br><span class="line">55</span><br><span class="line">56</span><br><span class="line">57</span><br><span class="line">58</span><br><span class="line">59</span><br><span class="line">60</span><br><span class="line">61</span><br><span class="line">62</span><br><span class="line">63</span><br><span class="line">64</span><br><span class="line">65</span><br><span class="line">66</span><br><span class="line">67</span><br><span class="line">68</span><br><span class="line">69</span><br><span class="line">70</span><br><span class="line">71</span><br><span class="line">72</span><br><span class="line">73</span><br><span class="line">74</span><br><span class="line">75</span><br><span class="line">76</span><br><span class="line">77</span><br><span class="line">78</span><br><span class="line">79</span><br><span class="line">80</span><br><span class="line">81</span><br><span class="line">82</span><br><span class="line">83</span><br><span class="line">84</span><br><span class="line">85</span><br><span class="line">86</span><br><span class="line">87</span><br><span class="line">88</span><br><span class="line">89</span><br><span class="line">90</span><br><span class="line">91</span><br><span class="line">92</span><br><span class="line">93</span><br><span class="line">94</span><br><span class="line">95</span><br><span class="line">96</span><br><span class="line">97</span><br><span class="line">98</span><br><span class="line">99</span><br><span class="line">100</span><br><span class="line">101</span><br><span class="line">102</span><br><span class="line">103</span><br><span class="line">104</span><br><span class="line">105</span><br><span class="line">106</span><br><span class="line">107</span><br><span class="line">108</span><br><span class="line">109</span><br><span class="line">110</span><br><span class="line">111</span><br><span class="line">112</span><br><span class="line">113</span><br><span class="line">114</span><br><span class="line">115</span><br><span class="line">116</span><br><span class="line">117</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">package</span> org.example;</span><br><span class="line"></span><br><span class="line"><span class="keyword">import</span> org.apache.hadoop.conf.Configuration;</span><br><span class="line"><span class="keyword">import</span> org.apache.hadoop.fs.FileSystem;</span><br><span class="line"><span class="keyword">import</span> org.apache.hadoop.fs.Path;</span><br><span class="line"><span class="keyword">import</span> org.apache.hadoop.io.LongWritable;</span><br><span class="line"><span class="keyword">import</span> org.apache.hadoop.io.Text;</span><br><span class="line"><span class="keyword">import</span> org.apache.hadoop.mapreduce.Job;</span><br><span class="line"><span class="keyword">import</span> org.apache.hadoop.mapreduce.Mapper;</span><br><span class="line"><span class="keyword">import</span> org.apache.hadoop.mapreduce.Partitioner;</span><br><span class="line"><span class="keyword">import</span> org.apache.hadoop.mapreduce.Reducer;</span><br><span class="line"><span class="keyword">import</span> org.apache.hadoop.mapreduce.lib.input.FileInputFormat;</span><br><span class="line"><span class="keyword">import</span> org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;</span><br><span class="line"></span><br><span class="line"><span class="keyword">import</span> java.io.IOException;</span><br><span class="line"></span><br><span class="line"><span class="keyword">public</span> <span class="class"><span class="keyword">class</span> <span class="title">PartitionerApp</span> </span>&#123;</span><br><span class="line"></span><br><span class="line">    <span class="comment">/**</span></span><br><span class="line"><span class="comment">     * Map：读取输入文件</span></span><br><span class="line"><span class="comment">     */</span></span><br><span class="line">    <span class="keyword">public</span> <span class="keyword">static</span> <span class="class"><span class="keyword">class</span> <span class="title">MyMapper</span> <span class="keyword">extends</span> <span class="title">Mapper</span>&lt;<span class="title">LongWritable</span>, <span class="title">Text</span>, <span class="title">Text</span>,</span></span><br><span class="line"><span class="class">            <span class="title">LongWritable</span>&gt; </span>&#123;</span><br><span class="line"></span><br><span class="line">        <span class="meta">@Override</span></span><br><span class="line">        <span class="function"><span class="keyword">protected</span> <span class="keyword">void</span> <span class="title">map</span><span class="params">(LongWritable key, Text value, Context context)</span> <span class="keyword">throws</span> IOException, InterruptedException </span>&#123;</span><br><span class="line">            String line = value.toString(); <span class="comment">//每一行的数据</span></span><br><span class="line">            String[] words = line.split(<span class="string">" "</span>); <span class="comment">//按空格 分隔符拆分</span></span><br><span class="line"></span><br><span class="line">            context.write(<span class="keyword">new</span> Text(words[<span class="number">0</span>]),</span><br><span class="line">                    <span class="keyword">new</span> LongWritable(Long.parseLong(words[<span class="number">1</span>])));</span><br><span class="line"></span><br><span class="line">        &#125;</span><br><span class="line">    &#125;</span><br><span class="line"></span><br><span class="line">    <span class="comment">/**</span></span><br><span class="line"><span class="comment">     * Reduce：归并操作</span></span><br><span class="line"><span class="comment">     */</span></span><br><span class="line">    <span class="keyword">public</span> <span class="keyword">static</span> <span class="class"><span class="keyword">class</span> <span class="title">MyReducer</span> <span class="keyword">extends</span> <span class="title">Reducer</span>&lt;<span class="title">Text</span>, <span class="title">LongWritable</span>, <span class="title">Text</span>,</span></span><br><span class="line"><span class="class">            <span class="title">LongWritable</span>&gt; </span>&#123;</span><br><span class="line">        <span class="meta">@Override</span></span><br><span class="line">        <span class="function"><span class="keyword">protected</span> <span class="keyword">void</span> <span class="title">reduce</span><span class="params">(Text key, Iterable&lt;LongWritable&gt; values, Context context)</span> <span class="keyword">throws</span> IOException, InterruptedException </span>&#123;</span><br><span class="line"></span><br><span class="line">            <span class="keyword">long</span> sum = <span class="number">0</span>;</span><br><span class="line">            <span class="keyword">for</span> (LongWritable value :</span><br><span class="line">                    values) &#123;</span><br><span class="line">                <span class="comment">//求key总次数</span></span><br><span class="line">                sum += value.get();</span><br><span class="line">            &#125;</span><br><span class="line">            <span class="comment">// 输出此次reduce统计结果</span></span><br><span class="line">            context.write(key, <span class="keyword">new</span> LongWritable(sum));</span><br><span class="line">        &#125;</span><br><span class="line">    &#125;</span><br><span class="line"></span><br><span class="line">    <span class="keyword">public</span> <span class="keyword">static</span> <span class="class"><span class="keyword">class</span> <span class="title">MyPartitioner</span> <span class="keyword">extends</span> <span class="title">Partitioner</span>&lt;<span class="title">Text</span>, <span class="title">LongWritable</span>&gt; </span>&#123;</span><br><span class="line"></span><br><span class="line">        <span class="meta">@Override</span></span><br><span class="line">        <span class="function"><span class="keyword">public</span> <span class="keyword">int</span> <span class="title">getPartition</span><span class="params">(Text text, LongWritable longWritable, <span class="keyword">int</span> numPartitions)</span> </span>&#123;</span><br><span class="line">            <span class="keyword">switch</span> (text.toString()) &#123;</span><br><span class="line">                <span class="keyword">case</span> <span class="string">"xiaomi"</span>:</span><br><span class="line">                    <span class="keyword">return</span> <span class="number">0</span>;</span><br><span class="line">                <span class="keyword">case</span> <span class="string">"huawei"</span>:</span><br><span class="line">                    <span class="keyword">return</span> <span class="number">1</span>;</span><br><span class="line">                <span class="keyword">case</span> <span class="string">"apple"</span>:</span><br><span class="line">                    <span class="keyword">return</span> <span class="number">2</span>;</span><br><span class="line">                <span class="keyword">default</span>:</span><br><span class="line">                    <span class="keyword">return</span> <span class="number">3</span>;</span><br><span class="line">            &#125;</span><br><span class="line">        &#125;</span><br><span class="line">    &#125;</span><br><span class="line"></span><br><span class="line">    <span class="comment">/**</span></span><br><span class="line"><span class="comment">     * 定义Driver：</span></span><br><span class="line"><span class="comment">     *</span></span><br><span class="line"><span class="comment">     * <span class="doctag">@param</span> args</span></span><br><span class="line"><span class="comment">     */</span></span><br><span class="line">    <span class="function"><span class="keyword">public</span> <span class="keyword">static</span> <span class="keyword">void</span> <span class="title">main</span><span class="params">(String[] args)</span> <span class="keyword">throws</span> IOException, ClassNotFoundException, InterruptedException </span>&#123;</span><br><span class="line"></span><br><span class="line">        Configuration configuration = <span class="keyword">new</span> Configuration();</span><br><span class="line"></span><br><span class="line">        <span class="comment">// 删除已存在的输出目录</span></span><br><span class="line">        Path outPath = <span class="keyword">new</span> Path(args[<span class="number">1</span>]);</span><br><span class="line">        FileSystem fs = FileSystem.get(configuration);</span><br><span class="line">        <span class="keyword">if</span> (fs.exists(outPath)) &#123;</span><br><span class="line">            fs.delete(outPath, <span class="keyword">true</span>);</span><br><span class="line">            System.out.println(<span class="string">"output file exists, but is has deleted"</span>);</span><br><span class="line">        &#125;</span><br><span class="line"></span><br><span class="line">        Job job = Job.getInstance(configuration, <span class="string">"wordcount"</span>);</span><br><span class="line"></span><br><span class="line">        <span class="comment">// 设置Job处理的类</span></span><br><span class="line">        job.setJarByClass(PartitionerApp<span class="class">.<span class="keyword">class</span>)</span>;</span><br><span class="line"></span><br><span class="line">        <span class="comment">// 设置作业处理的输入路径</span></span><br><span class="line">        FileInputFormat.setInputPaths(job, <span class="keyword">new</span> Path(args[<span class="number">0</span>]));</span><br><span class="line"></span><br><span class="line">        <span class="comment">// 设置map相关参数</span></span><br><span class="line">        job.setMapperClass(MyMapper<span class="class">.<span class="keyword">class</span>)</span>;</span><br><span class="line">        job.setMapOutputKeyClass(Text<span class="class">.<span class="keyword">class</span>)</span>;</span><br><span class="line">        job.setMapOutputValueClass(LongWritable<span class="class">.<span class="keyword">class</span>)</span>;</span><br><span class="line"></span><br><span class="line">        <span class="comment">// 设置reduce相关参数</span></span><br><span class="line">        job.setReducerClass(MyReducer<span class="class">.<span class="keyword">class</span>)</span>;</span><br><span class="line">        job.setOutputKeyClass(Text<span class="class">.<span class="keyword">class</span>)</span>;</span><br><span class="line">        job.setOutputValueClass(LongWritable<span class="class">.<span class="keyword">class</span>)</span>;</span><br><span class="line"></span><br><span class="line">        <span class="comment">// 设置job的Partitioner</span></span><br><span class="line">        job.setPartitionerClass(MyPartitioner<span class="class">.<span class="keyword">class</span>)</span>;</span><br><span class="line">        <span class="comment">// 同时还需要为job设置4个reduce task，每个partition一个</span></span><br><span class="line">        job.setNumReduceTasks(<span class="number">4</span>);</span><br><span class="line"></span><br><span class="line">        <span class="comment">// 设置作业处理的输出路径</span></span><br><span class="line">        FileOutputFormat.setOutputPath(job, <span class="keyword">new</span> Path(args[<span class="number">1</span>]));</span><br><span class="line"></span><br><span class="line">        System.exit(job.waitForCompletion(<span class="keyword">true</span>) ? <span class="number">0</span> : <span class="number">1</span>);</span><br><span class="line">    &#125;</span><br><span class="line">&#125;</span><br></pre></td></tr></table></figure></div>

<p>对于PartitionerApp，现要统计出不同品牌的手机总销量。假设输入文件in.txt，给出了手机品牌和对应的销量：</p>
<div class="highlight-box"autocomplete="off" autocorrect="off" autocapitalize="off" spellcheck="false" contenteditable="false"data-rel="PLAIN"><figure class="iseeu highlight /plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br></pre></td><td class="code"><pre><span class="line">huawei 200</span><br><span class="line">apple 180</span><br><span class="line">xiaomi 100</span><br><span class="line">huawei 150</span><br><span class="line">apple 30</span><br><span class="line">nokia 66</span><br><span class="line">meizu 66</span><br><span class="line">honor 88</span><br></pre></td></tr></table></figure></div>

<p>在Hadoop上运行PartiitonerApp之后，输出文件夹中会得到四个输出文件，因为在<code>MyPartitioner</code>类中指定了4个划分。这四个输出文件的文件名分别是<code>part-r-00000</code>,<code>part-r-00001</code>,<code>part-r-00002</code>,<code>part-r-00003</code>。</p>
<p>查看第一个输出文件<code>part-r-00000</code>，会得到第一个划分的结果：</p>
<div class="highlight-box"autocomplete="off" autocorrect="off" autocapitalize="off" spellcheck="false" contenteditable="false"data-rel="PLAIN"><figure class="iseeu highlight /plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">xiaomi 100</span><br></pre></td></tr></table></figure></div>

<p>查看第二个输出文件<code>part-r-00001</code>，会得到第二个划分的结果：</p>
<div class="highlight-box"autocomplete="off" autocorrect="off" autocapitalize="off" spellcheck="false" contenteditable="false"data-rel="PLAIN"><figure class="iseeu highlight /plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">huawei 350</span><br></pre></td></tr></table></figure></div>

<p>查看第三个输出文件<code>part-r-00002</code>，会得到第三个划分的结果：</p>
<div class="highlight-box"autocomplete="off" autocorrect="off" autocapitalize="off" spellcheck="false" contenteditable="false"data-rel="PLAIN"><figure class="iseeu highlight /plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">apple 210</span><br></pre></td></tr></table></figure></div>

<p>查看第四个输出文件<code>part-r-00003</code>，会得到第四个划分的结果：</p>
<div class="highlight-box"autocomplete="off" autocorrect="off" autocapitalize="off" spellcheck="false" contenteditable="false"data-rel="PLAIN"><figure class="iseeu highlight /plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><span class="line">honor 88</span><br><span class="line">meizu 66</span><br><span class="line">nokia 66</span><br></pre></td></tr></table></figure></div>

<p>从最后一个文件的结果可以看出来MapReduce对<code>key</code>默认根据字母进行了排序。honor &gt; meizu &gt; nokia 。</p>
<p>与wordcount相比，多了一个Patitioner Class：</p>
<div class="highlight-box"autocomplete="off" autocorrect="off" autocapitalize="off" spellcheck="false" contenteditable="false"data-rel="JAVA"><figure class="iseeu highlight /java"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">public</span> <span class="keyword">static</span> <span class="class"><span class="keyword">class</span> <span class="title">MyPartitioner</span> <span class="keyword">extends</span> <span class="title">Partitioner</span>&lt;<span class="title">Text</span>, <span class="title">LongWritable</span>&gt; </span>&#123;</span><br><span class="line"></span><br><span class="line">    <span class="meta">@Override</span></span><br><span class="line">    <span class="function"><span class="keyword">public</span> <span class="keyword">int</span> <span class="title">getPartition</span><span class="params">(Text text, LongWritable longWritable, <span class="keyword">int</span> numPartitions)</span> </span>&#123;</span><br><span class="line">        <span class="keyword">switch</span> (text.toString()) &#123;</span><br><span class="line">            <span class="keyword">case</span> <span class="string">"xiaomi"</span>:</span><br><span class="line">                <span class="keyword">return</span> <span class="number">0</span>;</span><br><span class="line">            <span class="keyword">case</span> <span class="string">"huawei"</span>:</span><br><span class="line">                <span class="keyword">return</span> <span class="number">1</span>;</span><br><span class="line">            <span class="keyword">case</span> <span class="string">"apple"</span>:</span><br><span class="line">                <span class="keyword">return</span> <span class="number">2</span>;</span><br><span class="line">            <span class="keyword">default</span>:</span><br><span class="line">                <span class="keyword">return</span> <span class="number">3</span>;</span><br><span class="line">        &#125;</span><br><span class="line">    &#125;</span><br><span class="line">&#125;</span><br></pre></td></tr></table></figure></div>

<p><code>getPartition(Text text, LongWritable longWritable, int numPartitions)</code>第一个参数是<code>key</code>，第二个参数是<code>value</code>，第三个参数<code>int numPartitions</code>是表示哪个partition的整数。</p>

          
      
    </div>
    
    
      <footer class="article-footer">
        完
      </footer>
    
  </div>
  
    
<nav id="article-nav">
  <div class="article-nav-block">
    
      <a href="../../27/Java-import-static/" id="article-nav-newer" class="article-nav-link-wrap">
        <strong class="article-nav-caption"></strong>
        <div class="article-nav-title">
          
            Java中的静态导入import static（导入类的静态成员）
          
        </div>
      </a>
    
  </div>
  <div class="article-nav-block">
    
      <a href="../../25/Linux-macOS-Shell%E7%BB%88%E7%AB%AF%E5%BF%AB%E6%8D%B7%E9%94%AE/" id="article-nav-older" class="article-nav-link-wrap">
        <div class="article-nav-title">Linux/macOS Shell终端快捷键</div>
        <strong class="article-nav-caption"></strong>
      </a>
    
  </div>
</nav>

    <!-- <link rel="stylesheet" href="/css/gitment.css">  -->
<script src='//unpkg.com/valine/dist/Valine.min.js'></script>

<div id="vcommentsContainer"></div>
<!--引用评论框输入特效js文件-->

<script src="../../../../js/shuru.js"></script>

<script>
  POWERMODE.colorful = true; // make power mode colorful
  POWERMODE.shake = true; // turn off shake
  document.body.addEventListener('input', POWERMODE);
</script>
<!-- valine评论系统 -->
<script>
  new Valine({
      el: '#vcommentsContainer',
      appId: 'AttjBiS7UwxkxjV1CSrNywoi-gzGzoHsz',
      appKey: 'F50aWycFFmAPLC8CfSW7rXdP',
      avatar: 'monsterid',
      enableQQ: true,
      placeholder: '昵称处填QQ号自动抓取网名和邮箱哟~'
  })
</script>


    <!-- <link rel="stylesheet" href="/css/gitment.css"> 
<script src="/js/gitment.js"></script>

<div id="gitmentContainer"></div>

<script>
var gitment = new Gitment({
  owner: '',
  repo: '',
  oauth: {
    client_id: '',
    client_secret: '',
  },
})
gitment.render('gitmentContainer')
</script>

 -->

  
  
</article>
</section>
        <aside id="sidebar">
  
    <div class="widget-box">
  <div class="avatar-box">
    <img class="avatar" src="https://q1.qlogo.cn/g?b=qq&amp;nk=979727728&amp;s=640" title="图片来自QQ"></img>
    <h3 class="avatar-name">
      
        许嵩老公
      
    </h3>
    <p class="avatar-slogan">
      深吸一口梦，吹个气球存起来。
    </p>
  </div>
</div>


  
    

  
    
  <div class="widget-box">
    <h3 class="widget-title">Tags</h3>
    <div class="widget">
      <ul class="tag-list" itemprop="keywords"><li class="tag-list-item"><a class="tag-list-link" href="../../../../tags/NET/" rel="tag">.NET</a></li><li class="tag-list-item"><a class="tag-list-link" href="../../../../tags/Cookie/" rel="tag">Cookie</a></li><li class="tag-list-item"><a class="tag-list-link" href="../../../../tags/ES6/" rel="tag">ES6</a></li><li class="tag-list-item"><a class="tag-list-link" href="../../../../tags/Git-Bash/" rel="tag">Git Bash</a></li><li class="tag-list-item"><a class="tag-list-link" href="../../../../tags/Github/" rel="tag">Github</a></li><li class="tag-list-item"><a class="tag-list-link" href="../../../../tags/Github-Actions/" rel="tag">Github Actions</a></li><li class="tag-list-item"><a class="tag-list-link" href="../../../../tags/HDFS/" rel="tag">HDFS</a></li><li class="tag-list-item"><a class="tag-list-link" href="../../../../tags/HTTP/" rel="tag">HTTP</a></li><li class="tag-list-item"><a class="tag-list-link" href="../../../../tags/Hadoop/" rel="tag">Hadoop</a></li><li class="tag-list-item"><a class="tag-list-link" href="../../../../tags/Hexo/" rel="tag">Hexo</a></li><li class="tag-list-item"><a class="tag-list-link" href="../../../../tags/Idea/" rel="tag">Idea</a></li><li class="tag-list-item"><a class="tag-list-link" href="../../../../tags/JVM/" rel="tag">JVM</a></li><li class="tag-list-item"><a class="tag-list-link" href="../../../../tags/Java/" rel="tag">Java</a></li><li class="tag-list-item"><a class="tag-list-link" href="../../../../tags/JavaScript/" rel="tag">JavaScript</a></li><li class="tag-list-item"><a class="tag-list-link" href="../../../../tags/Linux/" rel="tag">Linux</a></li><li class="tag-list-item"><a class="tag-list-link" href="../../../../tags/MapReduce/" rel="tag">MapReduce</a></li><li class="tag-list-item"><a class="tag-list-link" href="../../../../tags/Markdown/" rel="tag">Markdown</a></li><li class="tag-list-item"><a class="tag-list-link" href="../../../../tags/Maven/" rel="tag">Maven</a></li><li class="tag-list-item"><a class="tag-list-link" href="../../../../tags/Node/" rel="tag">Node</a></li><li class="tag-list-item"><a class="tag-list-link" href="../../../../tags/Node-js/" rel="tag">Node.js</a></li><li class="tag-list-item"><a class="tag-list-link" href="../../../../tags/SQL/" rel="tag">SQL</a></li><li class="tag-list-item"><a class="tag-list-link" href="../../../../tags/Serial-Port/" rel="tag">Serial Port</a></li><li class="tag-list-item"><a class="tag-list-link" href="../../../../tags/Shell/" rel="tag">Shell</a></li><li class="tag-list-item"><a class="tag-list-link" href="../../../../tags/SpringBoot/" rel="tag">SpringBoot</a></li><li class="tag-list-item"><a class="tag-list-link" href="../../../../tags/TCP/" rel="tag">TCP</a></li><li class="tag-list-item"><a class="tag-list-link" href="../../../../tags/VSCode/" rel="tag">VSCode</a></li><li class="tag-list-item"><a class="tag-list-link" href="../../../../tags/Vue/" rel="tag">Vue</a></li><li class="tag-list-item"><a class="tag-list-link" href="../../../../tags/WebSocket/" rel="tag">WebSocket</a></li><li class="tag-list-item"><a class="tag-list-link" href="../../../../tags/YARN/" rel="tag">YARN</a></li><li class="tag-list-item"><a class="tag-list-link" href="../../../../tags/hexo/" rel="tag">hexo</a></li><li class="tag-list-item"><a class="tag-list-link" href="../../../../tags/macOS/" rel="tag">macOS</a></li><li class="tag-list-item"><a class="tag-list-link" href="../../../../tags/ssh/" rel="tag">ssh</a></li><li class="tag-list-item"><a class="tag-list-link" href="../../../../tags/%E4%B8%AD%E6%96%87%E4%B9%B1%E7%A0%81/" rel="tag">中文乱码</a></li><li class="tag-list-item"><a class="tag-list-link" href="../../../../tags/%E5%A4%A7%E6%95%B0%E6%8D%AE/" rel="tag">大数据</a></li><li class="tag-list-item"><a class="tag-list-link" href="../../../../tags/%E5%AE%89%E8%A3%85%E6%95%99%E7%A8%8B/" rel="tag">安装教程</a></li><li class="tag-list-item"><a class="tag-list-link" href="../../../../tags/%E5%AF%86%E9%92%A5/" rel="tag">密钥</a></li><li class="tag-list-item"><a class="tag-list-link" href="../../../../tags/%E5%B9%B6%E8%A1%8C%E7%BC%96%E7%A8%8B/" rel="tag">并行编程</a></li><li class="tag-list-item"><a class="tag-list-link" href="../../../../tags/%E6%95%B0%E6%8D%AE%E5%BA%93/" rel="tag">数据库</a></li><li class="tag-list-item"><a class="tag-list-link" href="../../../../tags/%E7%94%B5%E8%84%91/" rel="tag">电脑</a></li><li class="tag-list-item"><a class="tag-list-link" href="../../../../tags/%E7%AB%B9%E4%BA%91/" rel="tag">竹云</a></li><li class="tag-list-item"><a class="tag-list-link" href="../../../../tags/%E7%BB%88%E7%AB%AF/" rel="tag">终端</a></li><li class="tag-list-item"><a class="tag-list-link" href="../../../../tags/%E8%84%9A%E6%9C%AC/" rel="tag">脚本</a></li><li class="tag-list-item"><a class="tag-list-link" href="../../../../tags/%E8%BD%AC%E8%BD%BD/" rel="tag">转载</a></li><li class="tag-list-item"><a class="tag-list-link" href="../../../../tags/%E8%BD%AF%E4%BB%B6/" rel="tag">软件</a></li></ul>
    </div>
  </div>


  
    
  <div class="widget-box">
    <h3 class="widget-title">Tag Cloud</h3>
    <div class="widget tagcloud">
      <a href="../../../../tags/NET/" style="font-size: 10px;">.NET</a> <a href="../../../../tags/Cookie/" style="font-size: 10px;">Cookie</a> <a href="../../../../tags/ES6/" style="font-size: 10px;">ES6</a> <a href="../../../../tags/Git-Bash/" style="font-size: 11.67px;">Git Bash</a> <a href="../../../../tags/Github/" style="font-size: 13.33px;">Github</a> <a href="../../../../tags/Github-Actions/" style="font-size: 10px;">Github Actions</a> <a href="../../../../tags/HDFS/" style="font-size: 10px;">HDFS</a> <a href="../../../../tags/HTTP/" style="font-size: 11.67px;">HTTP</a> <a href="../../../../tags/Hadoop/" style="font-size: 18.33px;">Hadoop</a> <a href="../../../../tags/Hexo/" style="font-size: 10px;">Hexo</a> <a href="../../../../tags/Idea/" style="font-size: 10px;">Idea</a> <a href="../../../../tags/JVM/" style="font-size: 18.33px;">JVM</a> <a href="../../../../tags/Java/" style="font-size: 20px;">Java</a> <a href="../../../../tags/JavaScript/" style="font-size: 16.67px;">JavaScript</a> <a href="../../../../tags/Linux/" style="font-size: 15px;">Linux</a> <a href="../../../../tags/MapReduce/" style="font-size: 11.67px;">MapReduce</a> <a href="../../../../tags/Markdown/" style="font-size: 10px;">Markdown</a> <a href="../../../../tags/Maven/" style="font-size: 13.33px;">Maven</a> <a href="../../../../tags/Node/" style="font-size: 10px;">Node</a> <a href="../../../../tags/Node-js/" style="font-size: 10px;">Node.js</a> <a href="../../../../tags/SQL/" style="font-size: 13.33px;">SQL</a> <a href="../../../../tags/Serial-Port/" style="font-size: 10px;">Serial Port</a> <a href="../../../../tags/Shell/" style="font-size: 11.67px;">Shell</a> <a href="../../../../tags/SpringBoot/" style="font-size: 10px;">SpringBoot</a> <a href="../../../../tags/TCP/" style="font-size: 10px;">TCP</a> <a href="../../../../tags/VSCode/" style="font-size: 10px;">VSCode</a> <a href="../../../../tags/Vue/" style="font-size: 10px;">Vue</a> <a href="../../../../tags/WebSocket/" style="font-size: 11.67px;">WebSocket</a> <a href="../../../../tags/YARN/" style="font-size: 10px;">YARN</a> <a href="../../../../tags/hexo/" style="font-size: 11.67px;">hexo</a> <a href="../../../../tags/macOS/" style="font-size: 11.67px;">macOS</a> <a href="../../../../tags/ssh/" style="font-size: 11.67px;">ssh</a> <a href="../../../../tags/%E4%B8%AD%E6%96%87%E4%B9%B1%E7%A0%81/" style="font-size: 10px;">中文乱码</a> <a href="../../../../tags/%E5%A4%A7%E6%95%B0%E6%8D%AE/" style="font-size: 10px;">大数据</a> <a href="../../../../tags/%E5%AE%89%E8%A3%85%E6%95%99%E7%A8%8B/" style="font-size: 10px;">安装教程</a> <a href="../../../../tags/%E5%AF%86%E9%92%A5/" style="font-size: 10px;">密钥</a> <a href="../../../../tags/%E5%B9%B6%E8%A1%8C%E7%BC%96%E7%A8%8B/" style="font-size: 10px;">并行编程</a> <a href="../../../../tags/%E6%95%B0%E6%8D%AE%E5%BA%93/" style="font-size: 10px;">数据库</a> <a href="../../../../tags/%E7%94%B5%E8%84%91/" style="font-size: 11.67px;">电脑</a> <a href="../../../../tags/%E7%AB%B9%E4%BA%91/" style="font-size: 11.67px;">竹云</a> <a href="../../../../tags/%E7%BB%88%E7%AB%AF/" style="font-size: 13.33px;">终端</a> <a href="../../../../tags/%E8%84%9A%E6%9C%AC/" style="font-size: 10px;">脚本</a> <a href="../../../../tags/%E8%BD%AC%E8%BD%BD/" style="font-size: 10px;">转载</a> <a href="../../../../tags/%E8%BD%AF%E4%BB%B6/" style="font-size: 10px;">软件</a>
    </div>
  </div>

  
    
  <div class="widget-box">
    <h3 class="widget-title">Archives</h3>
    <div class="widget">
      <ul class="archive-list"><li class="archive-list-item"><a class="archive-list-link" href="../../../../archives/2023/02/">二月 2023</a></li><li class="archive-list-item"><a class="archive-list-link" href="../../../../archives/2023/01/">一月 2023</a></li><li class="archive-list-item"><a class="archive-list-link" href="../../../../archives/2022/11/">十一月 2022</a></li><li class="archive-list-item"><a class="archive-list-link" href="../../../../archives/2022/09/">九月 2022</a></li><li class="archive-list-item"><a class="archive-list-link" href="../../../../archives/2022/08/">八月 2022</a></li><li class="archive-list-item"><a class="archive-list-link" href="../../../../archives/2021/12/">十二月 2021</a></li><li class="archive-list-item"><a class="archive-list-link" href="../../../../archives/2021/02/">二月 2021</a></li><li class="archive-list-item"><a class="archive-list-link" href="../../../../archives/2021/01/">一月 2021</a></li><li class="archive-list-item"><a class="archive-list-link" href="../../../../archives/2020/08/">八月 2020</a></li><li class="archive-list-item"><a class="archive-list-link" href="../../../../archives/2020/07/">七月 2020</a></li><li class="archive-list-item"><a class="archive-list-link" href="../../../../archives/2020/06/">六月 2020</a></li></ul>
    </div>
  </div>

  
    
  <div class="widget-box">
    <h3 class="widget-title">Recent Posts</h3>
    <div class="widget">
      <ul>
        
          <li>
            <a href="../../../../2023/02/13/%E4%BD%BF%E7%94%A8Node-js%E5%90%8E%E5%8F%B0%E8%AF%BB%E5%8F%96%E4%B8%B2%E5%8F%A3%E6%95%B0%E6%8D%AE%E5%B9%B6%E9%80%9A%E8%BF%87WebSocket%E5%AE%9E%E6%97%B6%E6%98%BE%E7%A4%BA%E5%9C%A8%E7%BD%91%E9%A1%B5%E4%B8%8A/">使用Node.js后台读取串口数据并通过WebSocket实时显示在网页上</a>
          </li>
        
          <li>
            <a href="../../../../2023/02/09/%E4%BD%9C%E4%B8%BA%E7%94%A8%E6%88%B7%EF%BC%8C%E4%BF%AE%E6%94%B9Cookie%E6%9C%89%E6%95%88%E6%9C%9F%EF%BC%8C%E5%BB%B6%E9%95%BFCookie%E6%9C%9F%E9%99%90/">作为用户，修改Cookie有效期，延长Cookie期限</a>
          </li>
        
          <li>
            <a href="../../../../2023/02/08/%E7%BB%88%E7%AB%AF%E5%91%BD%E4%BB%A4%E4%BB%A3%E7%90%86/">终端命令网络设置</a>
          </li>
        
          <li>
            <a href="../../../../2023/01/18/Windows%E7%B3%BB%E7%BB%9FC%E7%9B%98%E6%B8%85%E7%90%86%E5%B7%A5%E5%85%B7FreeMove%E5%8F%8A%E6%96%87%E4%BB%B6%E5%8D%A0%E7%94%A8%E8%BF%9B%E7%A8%8B%E6%9F%A5%E7%9C%8B%E8%BD%AF%E4%BB%B6LockHunter%E9%85%8D%E5%90%88%E4%BD%BF%E7%94%A8/">Windows系统C盘清理工具FreeMove及文件占用进程查看软件LockHunter配合使用</a>
          </li>
        
          <li>
            <a href="../../../../2023/01/14/AutoReconnectWiFi/">检测断网自动重连WiFi脚本</a>
          </li>
        
      </ul>
    </div>
  </div>

  
      <div class="widget-box">
    <h3 class="widget-title">友链</h3>
    <div class="widget">
      
        <!-- <a style="display: block;" href="../../../../https:/xiaoyan94.github.io/" title target='_blank' -->
        <a style="display: block;" href="https://xiaoyan94.github.io/" title target='_blank'
        >Github Pages</a>
      
        <!-- <a style="display: block;" href="../../../../https:/xy94.gitee.io/" title target='_blank' -->
        <a style="display: block;" href="https://xy94.gitee.io/" title target='_blank'
        >Gitee Pages</a>
      
    </div>
  </div>

  
 
  
</aside>
      </div>
      <footer id="footer">
  <div class="foot-box global-width center">
    &copy; 2023 许嵩老公 &nbsp;&nbsp;
    Powered by <a href="http://hexo.io/" target="_blank">Hexo</a>
    &nbsp;|&nbsp;主题 <a href="https://github.com/yiluyanxia/hexo-theme-antiquity" target="_blank" rel="noopener">antiquity</a>
    <br>
    <script async src="//busuanzi.ibruce.info/busuanzi/2.3/busuanzi.pure.mini.js"></script>
    <span id="busuanzi_container_site_pv">不蒜子告之   阁下是第<span id="busuanzi_value_site_pv"></span>个访客</span>
    
  </div>
  
  <!--引入文字点击特效-->
  <script src="https://code.jquery.com/jquery-2.0.3.min.js"></script>
  
<script src="../../../../js/dianji.js"></script>

</footer>

<!--添加回到顶部按钮-->
<style>
/* 小猫置顶 */
/* 自定义回到顶部样式 */
.cd-top {
  position: fixed;
  right: 25px;
  top: -900px;
  z-index: 99;
  width: 70px;
  height: 900px;
  /* background: url(https://cdn.jsdelivr.net/gh/moezx/cdn@3.1.9/img/Sakura/images/scroll.png) no-repeat center; */
  background: url(/images/scroll.png) no-repeat center;
  background-size: contain;
  -webkit-transition: all .5s ease-in-out;
  transition: all .5s ease-in-out;
  /* cursor: url(https://cdn.jsdelivr.net/gh/moezx/cdn@3.1.9/img/Sakura/cursor/No_Disponible.cur), auto; */
  cursor: url(/images/cursor/No_Disponible.cur), auto;
  opacity: 1
}

.cd-top.cd-is-visible {
  opacity: 1;
  top: -326px
}

.cd-top.cd-fade-out {
  opacity: 1
}

.cd-top:hover {
  opacity: 1
}

.cd-top span {
  display: none;
  color: #000;
  position: absolute;
  bottom: 0;
  height: 20px;
  width: 50px;
  text-align: center
}

@media screen and (max-width:860px) {
  .cd-top {
      display: none;
      height: 60px;
      width: 50px
  }
  .cd-top span {
      height: 10px;
      width: 50px
  }
}

#moblieGoTop {
  display: none;
  position: fixed;
  bottom: 10px;
  right: 10px;
  z-index: 99;
  border: 0;
  outline: 0;
  background-color: #fff;
  color: #404040;
  cursor: pointer;
  padding: 15px;
  border-radius: 10px;
  border-radius: 12px;
  box-shadow: 0 0 2px 0 rgba(0, 0, 0, .12), 0 2px 2px 0 rgba(0, 0, 0, .24);
  transition: box-shadow .2s ease
}
#moblieGoTop:hover {
  background-color: #fff;
  opacity: .8
}
.changeSkin-gear {
  position: fixed;
  bottom: 0;
  left: auto;
  right: 5px;
  width: auto;
  height: auto;
  z-index: 99;
  white-space: nowrap;
  padding: 10px 10px;
  cursor: pointer;
  border-radius: 10px 10px 0 0
}
</style>

<!-- 添加小猫置顶 -->
<a class="cd-top faa-float animated cd-is-visible cd-fade-out" style="top: -500px;"></a>
<button id="moblieGoTop" title="Go to top" style="display: none;font-size: xx-large;"><i class="fa fa-chevron-up" aria-hidden="true"></i>🔝</button>

<!-- <div class="scroll "> <i class="fa fa-arrow-up" style="margin-left: 4px;"></i>
Top⬆️
<span class="scrollpercent" style="margin-left: -2px;"></span>
<span style="margin-right: 4px; margin-left: -4px;">%</span>
</div> -->

<!-- https://www.huangpinke.com/2018/08/24/add-back-to-top-button.html -->
      <script src="https://code.jquery.com/jquery-2.0.3.min.js"></script>
<script>
if (!window.jQuery) {
var script = document.createElement('script');
script.src = "/js/jquery-2.0.3.min.js";
document.body.write(script);
}
</script>

  
<link rel="stylesheet" href="../../../../fancybox/jquery.fancybox.css">

  
<script src="../../../../fancybox/jquery.fancybox.pack.js"></script>




<script src="../../../../js/script.js"></script>




<script>
  (function(){
      var bp = document.createElement('script');
      bp.src = '//push.zhanzhang.baidu.com/push.js';
      var s = document.getElementsByTagName("script")[0];
      s.parentNode.insertBefore(bp, s);
  })();
  </script>

<!-- mermaid图 -->

  <script src='https://unpkg.com/mermaid@7.1.2/dist/mermaid.min.js'></script>
  <script>
    if (window.mermaid) {
      mermaid.initialize({theme: 'forest'});
    }
  </script>

    </div>
    <nav id="mobile-nav" class="mobile-nav-box">
  <div class="mobile-nav-img mobile-nav-top"></div>
  
    <a href="../../../../index.html" class="mobile-nav-link">首页</a>
  
    <a href="../../../../archives" class="mobile-nav-link">归档</a>
  
    <a href="../../../../quick-notes" class="mobile-nav-link">小抄</a>
  
    <a href="../../../../about" class="mobile-nav-link">关于</a>
  
  <div class="mobile-nav-img  mobile-nav-bottom"></div>
</nav>    
  </div>
</body>

</html>